Memorization in Deep Neural Networks: Does the Loss Function Matter?

Deep Neural Networks, often owing to the overparameterization, are shown be capable of exactly memorizing even randomly labelled data. Empirical studies have also that none standard regularization techniques mitigate such overfitting. We investigate whether choice loss function can affect this memorization. empirically show, with benchmark data sets MNIST and CIFAR-10, a symmetric as opposed either cross entropy or squared error results in significant improvement ability network resist then provide formal definition for robustness memorization theoretical explanation why losses robustness. Our clearly bring out role functions alone play phenomenon